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This resource book examines the value-added model 
approach when used in assessing early childhood Title I (ECI-I) 
programs.. The evaluation design must be able to separate prcgram 
effects from natural maturation- The basic idea behind the 
valu3-added model builds on the notion of natural maturation- The 
-f*l cr strengths are that it does not require a compariscn group or 
the use of a norm referenced test- The major weaknesses are its 
usefulness only for the assessment of skills or attributes which show 
a natural development with age over the duration of the program. 
Selection procedures may disguise the relationship between age and 
skill development among a particular group of program participants- 
Finally, it can require complex statistical calculations. This method 
attempts to derive a great deal of information from a situation with 
little data ana little external control. The validity of results from 
the value-added model may be questioned in situations where one 
wishes to assess the short-term impact of an ECT-I program and the 
available sample size is relatively small. A possible solution is 
pooling data across multiple years of the program or across several 
sites that are implementing similar activities, (DWH) 
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FOREWORD 

This booklet has been prepared as part of a project sponsored by the 
United States Education Department (USED) on evaluation in early childhood 
Title I (ECT-I) programs. It is one of a series of resource books developed 
in response to concerns expressed by state and local personnel about early 
childhood Title I programs. The series describes an array of diverse 
evaluation activities and outlines how each of these might contribute to 
improving local programs. The series revolves around a set of questions: 

• Who will use the evaluation results? 

• What kinds of information are users likely to find most helpful? 

• In what ways might this information aid in program improvement? 

• Are the potential benefits substantial enough to justify the cost 
and effort of evaluation? 

Together, the resource books address a range of issues relevant to the 

evaluation of early childhood programs for educationally disadvantaged 

children. The series comprises the following volumes: 

• Evaluating Title I Early Childhood Programs: An Overview 

• Assessment in Early Childhood Education 

• Short-Term Impact Evaluation of Early Childhood Title I Programs 

• An Introduction to the Value-Added Model and Its Use in Short-Term 
Impact Assessment 

• Evaluation Approaches: A Focus on Improving Early Childhood Title 
I Programs 

• Longitudinal Evaluation Systems for Early Childhood T?tle I Programs 

• Evaluating Title I Parent Education Programs 

Thr development of this series follows extensive field work on ECT-I 
programs (Yurchak § Bryk, 1979). In the course of that research, we 



identified a number of concerns that SEA and LEA officials had about ECT-I 
programs, and the kinds of information that might be helpful in addressing 
them. Each resource book in the series thus deals with a specific concern 
or set of concerns.. The books and the evaluation approaches they describe 
do not, however, constitute a comprehensive evaluation system to be uniformly 
applied by all. Our feasibility analysis (Bryk, Apling, § Mathews, 1978) 
indicated that such a system could not efficiently respond to the specific 
issues of interest in any single district at any given time. Rather, LEA 
personnel might wish to draw upon one or more of the approaches we describe, 
tailoring their effort to fit the particular problem confronting them. 

Finally, the resource books are not comprehensive technical manuals. 
Their purpose is to help local school personnel identify issues that might 
merit further examination and to guide the choice of suitable evaluation 
strategies to address those issues. Additional information and assistance 
in using the various evaluation strategies are available in the more techni- 
cal publications cited at the end of each volume, and from the Technical 
Assistance Centers in the ten national regions. 
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I. BACKGROUND AND RATIONALE 

Young children are constantly growing and changing over time even in 
the absence of an educational program. This grcwth may occur along the very 
dimensions that are the objectives of an early childhood Title I (ECT-I) 
program. Thus, in trying to assess the effects of any ECT-I program, the 
evaluation design must be able to separate program effects from natural 
maturation. Each of the basic USED evaluation ziodels can accomplish this, 
but in different ways. For example, with the ncrm-referencea model the 
adjustment for natural maturation is built into the noiming tables. For any 
given percentile, as the child grows older the raw score is assumed to rise 4 
accordingly.* 

The basic idea behind the value-added model builds directly on the 
notion of natural maturation end the fact that children of different ages 
will tend to display different test scores. Consider the sample of data 
shown in Figure 1. Notice that although there is considerable variation 
in the test scores on the Preschool Inventory measure, the scores tend to 
be higher for older children, IVe can summarize this relationship between 
t^st ^core and age by running a line through the scatter of points in such 
a way that the line is as close as possible, on average, to all of these 
points. Now there are many different lines tha: one could run though a 
scatter of points such as this. It is here tha: the statistical technique 



* Unfortunately, as described elsewhere (see tre resource book on Assess- 
ment in Early Childhood Education , Haney 6 Celberg, i?30), there are few 
adequate norm-referenced tests for use with ; oung children, and even where 
they exist, the norms may not be appropriate for use in assessing the 
short-term impact of an £CT-I program. In such situations, the value- 
added model described in this booklet and the other strategies discussed 
by Haney are possible alternatives. 
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or regression analysis is particularly helpful in determining a good choice 
of a single line. The resulting equation, called the regression line, for 

predicted test score = -13.9 + .48 x child's age* 

This equation is a statistical description of the relationship between test 
score and age. Given the age of any child, we can use this equation to 
predict a test score for that child. For example, if a child were 4 years 
and 7 months old (55 months) our equation would yield: 



predicted test score = -13.9 + .48 (55) 
predicted test score = -13.9 + 26.4 = 12.5. 



Preschool Inventory 
Measure (Number of 
Items Correct) 
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Figure 1 . Sc atterplot of PSI Pretest by Age (Head Start 
F 1 anned Variation Evaluation") . 

Source: Anderson et al . , 1980. 
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Note in Figure 1 that the test "scores of children 55 months old tend to 
center around 12.5, although some scores are considerably different. Similar 
ly, if a child were 5 years old (60 months) our equation would yield: 

predicted test score = -13.9 + .48 (60) = 14.9. 

Thus, for older children the equation predicts a somewhat higher test score. 
So our regression (also called prediction) equation captures a fundamental 
^fltur? of thc^e data — test scores tend to be higher on average for older 
children, although clearly not in every case. 

The scatterplot and regression line in Figure 1 is a formal way of 

9 

representing this maturation phenomenon. In particular, the feature of 
growth over time is captured in what is called the regression coefficient 
for age or the slope of the regression line. In the equation above, the 
slope is .48, and it represents the expected gain per month in test score 
points for any child. We can think of this as the natural growth rate for 
this group of children in the absence of any special program. For these 
data, we can expect children to gain, on average, about a half a test score 
point per month (to be precise, .43 points per month) as a result of natural 
maturation . 

The value-added model builds directly on the relationship between test 
score and age and the information it contains about natural growth. Using 
thf> slope fron the regression of pretest scores on age, we can project the 
growth that children in an ECT-I program would have attained in the absence 
of that program. This projection serves as the "no-treatment" expectation. 
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The observed or actual gain under the program is compared with the projected 
growth under natural maturation, The difference between the two provides 
an estimate of the effect of the ECT-I program, or what we refer to as the 
value added by the program. 

One of the major differences between the value-added strategy and the 
other four models for examining the short-term impact of ECT-I programs* 
is the standard of comparison used for assessing the effects of the program. 
The norm-referenced model relies on test norms to generate this comparison 
value. The control-group roooel uses test data from a group of children 
who are similar to the Title I recipients but are not receiving Title I 
% services. The special regression model is a variant on the control-group 
approach. In criterion-referenced approaches the standard of comparison 
is set by professional judgment. The value-added model, in contrast, does 
not require a control group, norms, or an externally set criterion. Rather, 
the relationship of age with pretest scores is used to estimate the natural 
growth that might have resulted if no special program had been offered during 
the interval between the pre- and posttest. 
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* See the resource book on Short-Term Impact Evaluation of Early Child - 
hood Title I Programs, Haney (1980) . ~ 



II. THE BASIC VALUE-ADDED MODEL 

We introduce the basic value-added model through an example. The data 
for this illustration are drawn from a 1977-78 Title I kindergarten program 
in Iowa. As part of a larger evaluation effort, the staff administered the 
Boehm Test of Basic Skills as a pretest in the early fall and again as a 
posttest late in the spring. The actual duration between pre- and postt-st ' 
was 7 months. By examining the relationship between the pretest scores and 
children's ages, we can develop an estimate of the natural growth rate in 
the absence of the program, and use this to estimate the expected gain under 
natural maturation. 

Figure^/ displays the relationship between pretest score and age for 
these data. Again, the raw scores tend to be higher for older children. 
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-igure 2. ' Scatterplot of Boehm Pretes t by Age 
(Iowa ECT-I Data). 
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Through a simple regression of the Boehm pretest raw scores on age 
(measured in months at the time of the pretest), we estimate a -natural 
growth rate (the slope of the regression line) of .47 points per month. 
In other words, the estimated average growth rate for these children in 
the absence of the Title I program is about half a raw scoTe point per 
month on the Boehm, 

Assuming that the children would continue to mature naturally at the 
same average rate even without an especially effective program, we can now 
estimate the expected gain over the course of the program interval due 
solely to natural maturation: 



expected gain 
(due to 
maturation) 



natural growth 
rate per month 
(slope of re- 
gression line) 



program 
duration 
(measured 
in month<) . 



\ 



For the Iowa Title I kindergarten program, 



rxpected gain = 



,47 raw score 
points 
par month 



x / months = 3.29 points 



To estimate the program effect, :e compare the expected gain with the 
actual observed gains of children in the program. For the Iowa program, 
the pretest mean was 21.27, and the posttest mean 30.47, yielding an ob- 
served gain (posttest mean jiiinus pretest mear.) of 9.20 points. The estimated 
short-term effect, or value added by the propa-, is simply 



value added 



observed gain 



expected ga4n, 
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which for our illustration yields 



value added 

byECTI = 9.20 - 3.29 = 5.91 points, 

program 



If we compare the estimated value added by the program to the expected 
gain under maturation, we have a natural way to assess the educational 
significance of the program effect estimate: 



index of 

educational = (value added / expected gain) x 100 . 
significance 



For the Iowa data this yields 



index of 

educational 

significance 



C 5.91 / 3.29 ) x 100 = 180%. 



In this case, the value-added model estimates that the program produced a 
180% improvement in the average growth over what would have been expected 
in the absence of the program. 

Tnus, the basic value-added model only requires information on each 
child's pretest and posttest score, the age at pretest, and the duration 
between pre- and posttest points. Neither a control group nor information 
on background variables (e.g. demographic characteristics and home environ- 
ment) is required. If the latter is available, however, more precise program 
effect estimates are possible, and there is also an opportunity to estimate 
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program effects for different subgroups of children (e.g. boys vs girls), 
Ine next section presents an illustration of this extension of the value- 
added model , 

As is true of all of the short-term impact evaluation models for ECT-I 
programs, the validity of the program effect estimated by the value-added 
model depends upon some basic assumptions. For the value-added approach, 
the key concern is our ability to derive an estimate of the natural growth 
rate from the observed relationship between children's pretest scores and 
ages. The estimate derived from the regression of the pretest on age is 
valid only if we can assume that the average growth rate is stable across 
children of different ages within the program group (e.g., the older children 
cannot be the "slow developers"), and that the average growth rate will 
remain stable over the duration of the program. We refer to these conditions 
as the stable universe assumption. In a later section, we discuss some of 
the ways in which this assumption might be violated in ECT-I programs, and 
the consequences of such violations. 
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III. THE VALUE-ADDED' MODEL WITH BACKGROUND VARIABLES 

The basic val ue-added model estimates an average growth rate for all 
children that would be expected to occur in absence of the ECT I program, 

T t i C roocAno^l o + r\ nccn**r* V> 4-U » +■ *-U r> »i »» ♦ 1 ^ ~ ^ „ 

— - — — * * w *.w M^^uiMV) uvnv ir c a , tiiOv utU iio.cuj.cij. gxvnut i at& voilCi 

across children, and that children with different background characteristics 
(e.g. boys and girls) might have different average growth rates. Thus, it 
seems natural to extend the value-added model, when background information 
is available, to estimate different average growth rates for children with 
different background characteristics. 

We introduce this extension of the model with a hypothetical example. 
Let's assume that in evaluating some ECT-I programs we had data on the child's 
sex in addition to the basic data on pretest, posttest, age, and program 
duration. The first step in the analysis would be to plot the pretest-age 
relationship separately for boys and girls, and to perform separate regres- 
sions of the pretest on age for each group. This is illustrated in Figure 3. 
The regression equation for girls might be: 

predicted est score = -45.0 + 1.1 x age in months 
and for boys ; 

predicted test score = -36.0 + 0.9 x age in months . 

Thus, we now have different natural growth rate estimates (i^e. different 
regression slopes) for girls (1.1 points per month) and for boys (0.9 points 
per month), he can use these estimates to compute separate program effect 
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Figure 3. Scatterplots of Pretest by Age, Separately 
fc r Girls and Boys (Hypothetical Data) . 
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estimates for boys and girls arid an overall estimate* For girls, 



expected gain * 1.1 points per month x 8 months = 8.8 points, 

(program 
duration) 

and for boys, 



expected gain = .9 points per month x 8 months = 7.2 points. 

To estimate the program effect, we compare these expected gains with 
the actual observed gains. The mean test scores for boys and girls might 
be something like this: 



Girls 



Boys 



Posttest means 30.0 22.0 

Pretest means 15,0 5.0 

Observed gains 15,0 17.0 

Expected gains 8.8 7.2 

Value-added 
(observed gains 

minus expected 6.2 9.8 
gairs) 



Index of Educational 
Significance 



70.5% 136. l c < 



Thus, the program seems to have had a somewhat larger effect for boys than 
for girls. In .rder to develop an overall estimate, ve weight each of the 
separate estimates proportionally to that group's representation in the 
program. For example, if the program group consisted of 100 children-- 
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30 girls and 70 boys--then the composite value-added estimate would be 
overall 

value-added = (30/100) x 6.2 + (70/100) x 9.8 = S.7 points, 
estimate 

* 

Thus, the average effect of the program is 8.7 points. 

While this example serves to illustrate the basic mechanise for in- 
corporating background information into the value-added model, this approach 
becomes computationally unwieldly when we have multiple background variables, 
and when some of these are continuous--e.g. , information on fanily income 
measured in dollars. While it is possible to take a continuous variable 
and create groups or categories (e.g. low, middle, and high income families), 
this is not a very efficient way to use the information . ^ftsther* we can 
approach this as a multiple regression analysis problem. Appendix A presents 
a complete mathematical model, and a worked example of this approach, Here,^ 
we present only the simplest case using one background variable (agair. in- 
formation on child's sex). 

We can think of a growth rate for each child in the absence of an ECT-I 
program as consisting of a base, or average, growth rate plus adjustnents 
to the base associated with particular background characteristics of the 
child. In the basic value-added model, the regression equazic?* had or.ly 
one independent variable: age in months. Now, we add to the equatior. other 
independent variables to represent the additional pieces of background in- 
formation. In particular, we might want to know how sex grcup membership 
alters the relationship between pretest ana ?ge. We refer to this as the 
interaction of sex with age, and we represent it in the regression equation 
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as follows. We begin by designating some number to represent each sex group 
The choice of number is totally arbitrary, but it is often convenient to 
use 1.0 for girls and -1.0 for boys. We can think of this as an indicate- 
variable that identifies the sex of the child (values of 1 for girls, -1 f or 
boys."). Next, wp mulfinlv aof> hv tho coy inrii , MT ^ki„ _ 

by-age interaction variable.* We then regress the pretest scores on age and 
on the age x sex interaction variable to determine a new prediction equation 
that contains more information on natural growth rate. Although the actual 
computational formulas are somewhat complicated, the results are fairly in- 
tuitive. 

We illustrate the technique with data from a short-term impact evaluation 
of an ECT-I first-grade program in Rhode Island. Each child was pre- and 
posttested on the Peabody Picture Vocabulary test. Regressing the pretest 
scores on age and the age x sex interaction variable yielded: 



predicted test score = 17.5 + .56 x age - .05 x age-by-sex interact! 



on. 



It can be shown easily that the regression slope for age (.56) represents 
the base growth rate, and that the regression slope for the interaction term 
(.05) represents the adjustment to the growth rate associated ^ith sex group 
membership . 



This technique can be extended for background variables that have more 
than two categories. In this case, ue would create a series of indicator 
variables to represent membership in the various possible groups. Alter- 
natively, if the background variable is continuous, we do net create in- 
dicator variables, but directly multiply the background variable and age 
to create the interaction variable. More details on this technique can 
be found in a text on applied regression analysis such as j. Cohen and 
p - Cohen. Applied Multiple Regression/Correlational Analyses for the 
Be havioral Sciences . Hillsdale, N.J. : Lawrence Erlbaum Associates ,*" 1975 . 

ERIC ij 



-14- 



For the Rhode Island data, 



natural growth rate for girls = .56 - .05 (1.0) = .51 points per month 

t 

value indicating 
"girl" 

natural growth rate for boys = .56 - .05 (-1.01 = .61 points per month. 

t 

value indicating 
"boy"* 



The average program duration was 4.6 months for girls and 4.7 months for 
boys. Thus, for girls, 



expected ?ain = .51 points per month x 4.6 months =2.3 points, 



and for boys, 



expected gain = .61 points per month x 4.7 months = 2.9 points. 



* While the choice of the values 1 and -1 is arbitrary, the size of the 
adjustment (i.e. the regression slope for the interaction variable) 
will depend on the particular values chosen. The separate growth rate 
estimates for girls and boys, however, will remain the same regardless 
of the choice of values for the indicator variable. For example, if 
we had chosen 2 for boys and ~2 for girls the following prediction 
equation Would have resulted: 

predicted test score = 17.5 + .56 x age + .025 x age-by-sex interaction 
This equation, however, yields the same growth rates: 

natural growth rate for girls = .56 + .025 (-2) = .51 

value indicating 
"girl" 

natural growth rate for boys = .56 + .025 (2) = .61. 

t . J . . 
value indicating 

"boy" 
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The observed test scores for the Rhode Island ECT-I kindergarten program 
were: 

Girls (n=21) Boys (n=34) 
(total sample size = 55) 



Posttest means 


53.8 


59.4 


Pretest means 


50.5 


57.5 


Observed gains 


3.3 


1.9 


Expected g^inT\ 


2.3 


2.8 


Value added 


1.0 


-0.9 


Index of Educational 
Significance 


43% 


-32% 



Overall value added = (21/55) 1.0 + (34/55) (-.9) = -.17. 

These results suggest a small positive program effect for girls, and a 
corresponding small negative effect for boys. On average, there appears 
to be no significant program effect. 
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IV. DISCUSSION OF MODEL ASSUMPTIONS 

The idea underlying the value-added strategy is to use information 
about natural growth contained in the relationship between pretest and age 
(and background variables) to predict an expected gain in the absence of 
the ECT-I program. This expected gain represents the standard against 
which we compare the observed gain, and thereby assess the program's effec- 
tiveness. Thus, this approach requires neither a comparison group nor the 
use of a norm-referenced test in order to develop its standard of comparison, 
The validity of the approach, however, rests on an important assumption- - 
the stable universe— which is key to our ability to estimate the expected 
growth in the absence of the program from the pretest and age relationsh p. 

ASSUMPTION OF A STABLE UNIVERSE 

The assumption of a stable universe is basic to any attempt to draw 
longitudinal inferences (e.g. expected growth for ECT-I children over a 
period of time) from cross-sectional data [e.g. test data such as pretest 
collected at one point in time). In particular, the value-added model 
assumes that the score at the pretest point represents the cumulative ef- 
fect of natural growth up to that time point. While the influences of 
other factors (e.g. test-situational effects) are also represented in the 
observed test scores, the model requires that such influences be independent 
of the child's age. 

Problems* with this assumption can arise in several ways. First, there 
may be historical trends or accidents causing children born at different 
times to differ. For example, children born after a major outbreak of 
rubella might on average have somewhat lower growth -rates. The model 
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assumes, however, that children bom in say January of 1978 will grow at 
approximately the same average rate as children born in June 1978. For 
relatively homogeneous age groups (i.e., where the age range of children 
in the program groir is less than 24 months), historical trends or accidents 
are unlikely to be a problem. 

Second, schooling experiences prior to ECT-I programs can also be a 
source of concern. Since older children are more likely to have had such 
experiences, this pretest-age relationship may confound both natural growth 
influences and any effects of prior programs. Estimating expected gains on 
the basis * such data can be a serious misrepresentation, and as a result 
yield a very biased estimate of the program effect. Thus, application of 
the value-added model should generally be limited to the youngest groups (e.g., 
those in prekindergarten and kindergarten programs), who are most likely to 
have received little or no prior formal schooling. The more the ECT-I 
participants vary in previous school experiences, the more strain is placed 
on the stable universe assumption. 

Third, the process of selecting children for the program may introduce 
a problem. For example, the oldest children in an ECT-I preschool might 
be delayed entrants into the school group, and the youngest somewhat more 
precocious than average. Such cases might show up as outliers in a pretest 
by age scatterplot (see Figure 4). If only a few cases like this occur in 
tne program sample, they can be set aside and the value-added analysis ap- 
plied to the remaining cases. For any case deleted as an outlier, there 
should be some corroborating evidence, such as teacher reports or parent 
interviews, to document that in fact this case is unusual. If the problem 
is more widespread, however, application of the value-added model is not 
appropriate. 

23 
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Figure 4. Differential Program Selection of Children by Age 
(Hypothetical DataX 

LINEAR GROWTH ASSUMPTION 

In using the simple regression of pretest on age to estimate the natural 
growth in the absence of an ECT-I program, we are assuming that the average 
natural growth trajectory is linear--that is, that the expected gain per 
month is independent of the age of the child (see Appendix A for more details). 
IvTiile this may be too simplistic a model to describe many educational processes 
in any detail, it should often be a reasonable approximation over a short 
period such as a six- to eight-month program. Over longer durations it is 
less likely to be adequate and use of the irodel should be approached with 
caution. 
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Now a violation of either the stable universe or the linear growth as- 
sumption will usually evidence itself in the pretest and age scatterplot. 
We have already seen in Figure 4 some data where differential selection by 
age had occurred . In general, if the linear growth and stable universe 
assumptions are valid, we should see pretest and age scatterplots similar 
to those in Figures 1 and 2. The line that we place through the scatter 
of points does a good job of summarizing ihe basic relationship between test 
score and age. Compare this with the situation represented in Figure 5. 
Here, the test score and age relationship is not well summarized by a line, 
but rather requires a curve to represent it. Thus, a partial test of the 
appropriateness of the value-added model is whether the pretest and age 
relationship appears linear. A further consideration is whether the pre- 
test and age scatterplot appears as a tipped funnel (see Figures 1, 2, and 
6). Under most situations involving linear individual growth, ve expect 
test scores to be more variable for older children. This is reflected in 
the scatterplot by the widening of the funnel associated with older ages. 

If a violation of either the linear growth cr the stable universe as- 
sumption occurs, it will usually be reflected in the pretest and age scatter- 
plots. Tipped funnels such as those found in Figures 1, 2, end 6 are very 
unlikely in such situations. While the existence of a tipped funnel scatter- 
plot doesn't assure validity of the value-added model, it is a useful empirical 
tool for identifying possible problems. We have already indicated how dif- 
ferential selection effects may be identified and partially compensated for. 
More generally, if one suspects nonlinear individual growth and finds a 
nonlinear scatterplot, such as in Figure 5, 5t is often possible to transform 



9 



.21. 



Pretest 



25 -- 



20 — 



IS -- 



10 -- 



5 - 



50 



• •• 



Age (months) 



65 



Figure 5- Nonlinear Relationship between Pretest Score and Age 
(Hypothetical Data) , 





Figure 6. Pretest vs. Age: Tipped Funnel Scatterplot 
(Hypothetical Data ) . 
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the test score into an alternative metric C e »g» logarithmic or negative 
exponential transformation) in which the test-score/age relationship is 
again linear. The interested reader is referred to Bryk, Strerio, and 
Weisberg, 1980, for more details and an illustration. 

One should, however, approach such transformation with caution. The 
existence of a nonlinear pretest and age scatterplot does not prove that 
individual growth is nonlinear. Violation of the stable universe assumption 
can also give rise to scatterplots which are nonlinear in appearance even 
though individual growth may be well represented by a straight line. Thus, 
if we observe a nonlinear scatterplot, we should first look carefully at 
the ECT-I evaluation design for possible violations of the stable universe 
assumption before going ahead with some nonlinear transformation. 

t 

ASSORTED OTHER ASSUMPTIONS f 
In varying degrees, each application of the value-added model requires 
some extrapolation beyond the available data. We can think of the value- 
added model as an approach to predicting an expected average test score for 
a group of children at some future time. The model implicitly assumes that 
the relation of age to test score apparent at pretest time will still be 
valid at posttest time when the children are significantly older. For example, 
if ECT-I participants are 45 to 57 months old at pretest time and the progran 
lasts nine months, we must assume that the natural growth rates estimated 
at the pretest time point extrapolate into the age range of 54 to 64 months. 
Such extrapolation can sometimes be troublesome. Ideally, the age range 
within the sample should be considerably larger than the expected program 
duration. For example, if we were evaluating the effects of a six-month 
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^ preschool program, it would be desirable to have a spread of 12 to 18 months 
in children's ages. By contrast, in situations where the program duration 
exceeds the age range in the sample, the value-added approach is generally 
not recommended. I 

Finally, the model assumes that the outcomes to be evaluated show a 
natural increase with age across the duration of the program. For some 
variables this may not be the case. For example, an ECT-I program might use 
an ability test, e.g. the Slossen, as a program outcome. In the standard 
metric (mean equals 100, standard deviation of 15), such tests do not 
normally display any systematic relationship to age since the purpose of 
the standardization is to remove them. The value-added model can be applied 
in such cases by. merely transforming the scores back into a mental age metric. 
In other cases, however, application of the value-added model may simply be 
inappropriate. By the first grade, for example, most children have developed 
the gross motor skills involved in skipping or running, so measures of gross 
motor skills would show little if any relationship with age in this range. 
Application of the value-added model in such a case might yield an expected 
gain of zero or even a negative amount. The validity of this as a standard 
for comparison vis-a-vis the observed gain is subject to considerable question. 
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V. PROBLEMS LIKELY TO BE ENCOUNTERED IN APPLICATION TO ECT-I PROGRAMS 

V 

In developing this resource book we had an opportunity to apply the 
value-added model to several existing ECT-I data sets. The samples in 
previous sections were drawn from these analyses. Since the value-added 
approach has not seen widespread use, these analyses were helpful in iden- 
tifying new problems, particularly those that might commonly affect ECT-I 
applications. 

First is the familiar problem of testing and instrumentation. Floor 
and ceiling effects on either pretest or posttest can cause special dif- 
ficulties in the value-added approach, because it depends upon the assumed 
stable relationship between age and test scores for all children. A floor 
or ceiling effect on the pretest will obscure the natural growth relation- 
ship of test scores with age, and thus interfere with our ability to estimate 
expected gains in the ab sence of the p rogram. Similarly, suc h effects on 



the posttest would make it difficult for the program to exhibit a positive 
effect no matter how worthwhile the effort. In short, the value-added model, 
like the other ECT-I short-term impact models, requires good instrumentation 
if the evaluation is to yield useful results. 

Second, if selection for the ECT-I program is based on the pretest 
score or some other test score highly correlated with it, a routine ap- 
plication of the value-added model will not be valid. Figure 7 illustrates 
the scatterplot of pretest score and age for one of the ECT-I data sets that 
we analyzed. This LEA apparently used a cutoff score of 46 on the pretest 
in selecting children for the ECT-I kindergarten program. Figure 7 clearly 
shows how the cutoff rule eliminates any relationship that may normally 
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exist between age and pretest. As shown 
of pretest on age will underestimate the 
overestimate the program effect, in such 
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Figure 7. Scatterplot of Pretest Score and Age 
Under a Selection Cutoff Rule. 



Now, if the program sample was drawn from a larger group through the 
use of a cutoff score rule, and if pretest and age data exist on this larger 
sample of children, then the basic natural growth can still be estimated by 
iegressing the pretest scores on age for the entire group. The slope from 
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this regression should provide us with an unbiased estimate of the natural 
growth rate, which can be used to compute an expected gain and value added 
for the program sample. Alternatively, if data on the larger group do not 
exist, it is still possible to estimate the average natural growth rate, 
though the simple regression of pretest on age no longer suffices. Using 
more complex statistical methods, discussed in Appendix B, it is possible, 
although more difficult, to estimate the natural growth rate from the trun- 
cated data. 

Third, since the value-added model hinges on this estimation of a slope 
or regression coefficient for the pretest and age relationship, its applica- 
tion with small sample sizes should be viewed with caution. With a small 
sample, the presence of just one or two outliers can significantly distort 
our estimate of the natural growth rate (i.e. the regression coefficient 
or slope). For sample sizes smaller than 30, even the basic value-added 

^aodel^^ i. e . th e simp4e -regression-of pretest -on age) m can be quite i t ive 

to sampling variations. As for application of the value-added model with 
background variables, this should be limited to fairly large sample sizes. 
If we wish to apply the model separately for distinct subgroups (e.g- boys 
and girls), then it least 30 subjects per group would be desirable. If we 
wish to apply the regression approach with several age-by-background inter- 
action terms, then even larger sample sizes would be required. 

Thus, in situations where we wish to assess the short-term impact of an 
ECT-I program and the available sample size is fairly small, the validity of 
results from the value-added model may be open to question. In these situa- 
tions, pooling of data across multiple years of the program, or perhaps across 
several sites that are implementing similar activities, represents a possible 
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solution. An alternative formulation of the value-added model, employing an 
empirical Bayes approach to estimating the natural growth rate, appears par- 
ticularly promising in dealing with data from multiple sites or years,* 



\ 



* The algorithm for this method, however, is somewhat complicated, and requires 
some special computer programming. A discussion of the technique and illus- 
trations can be found in J. Strenio, Empirical Bayes Estimation for a Hier- 
archical Linear Model (doctoral dissertation, Harvard University, 1981) . 
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VI. CONCLUDING REMARKS 

The value-added approach to measuring short-term impact of ECT-I 
programs has both strengths and weaknesses. Its major strengths are that 
it does not require a comparison group or the use of a norm-referenced 
test. Its major weaknesses are that (1) it is appropriate only for the 
assessment of skills or attributes which show a natural development with 
age over the duration of the program; (2) selection procedures may disguise 
the relationship between age and skill development among a particular group 
of program participants (thus either precluding application of the value- 
added approach or necessitating reliance on some additional data as a source 
of deriving appropriate age/skill-development projections); and (3) it can 
require some quite complex statistical calculations. 

We should view each application of the value-added model with some 
reservations. We are attempting to derive a great deal of information (i.e. 



an estimate of the program effect) from a situation with little data (i.e. 
no comparison group or valid norm) and little external control (i.e. no 
random assignment). While it is possible to develop an estimate of the 
program effect, its validity should be carefully examined along lines sug- 
gested in the last two sections. 

More generally, as the very first test, we should always ask the question 
"Does it make sense?" For a variety of reasons, mentioned above, it is 
quite possible that in an individual application the pretest and age relation- 
ship might appear negative, implying an average loss in test score points 
in the absence of a program. In most situations, expectations of negative 
gains due to natural maturation would be nonsensical, and the evaluator 
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should discard the analysis as clearly incorrect. Si^i.larly, it is quite 
possible in an individual application that the expected gain under the value 
added model, when added to the average pretest score, exceeds the maximum 
score for this test or instrument. If such a ceiling effect appears, applic 
tion of the value-added model is again inappropriate. 

Finally, only so much data analytic advice can be packaged in a fairly 
short and nontechnical resource booklet. It cannot substitute for technical 
expertise fully grounded in an understanding of the statistical model and 
its estimation procedures. Whenever possible, such professional assistance 
should be sought to help in the application of this approach. 



APPENDIX A 

TECHNICAL DISCUSSION: 

(1) BASIC VALUE-ADDED MODEL 
(2) EXTENDING THE MODEL TO INCORPORATE BACKGROUND VARIABLES 



This appendix is a more technical introduction to the value-added approach. 
It summarizes Bryk, Strenio, and Weisberg (1980), and the interested reader 
is referred there for more details. 
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BASIC VALUE-ADDED MODEL 

The value-added model focuses on the natural growth of subjects prior 
to an ECT I program, attempting to project explicitly a posttest status for 
the program group as if they had been subject to the control condition. The 
actual growth is then compared with projected growth, the difference re- 
presenting the effects of the program. 

The model assumes that over the duration between pre and posttest each 
individuals growth consists of two components: (1) systematic growth, which 
can be characteristized by a growth rate and a corresponding growth curve; 
and (2) an individual noise or random component, which is specific to a 
particular subject at a certain point in tims. Thus we can represent the 
observed score for individual i at any time t as 

Y.(t) « G.(t) + R.Ct), (1) 

where G^(t) represents systematic growth and R^(t) represents the rar.dom 
component . 

The i::~L . idual ! s systematic growth, G^(t), is represented as a function 
of age (or some other time metric). While in principle this function may 
take any form, it may often be adequate to assume that it is linear; 

G.(t) = Tr.a.OO + & i$ (2) 

where tk represents the slope, 6^ represents the Y intercept, and a^t) fs 
the age for subject i at time t. Individuals may vary in terras of a growth 
rate, tt, and an intercept parameter, 6. The model assumes that tt anc. 5 are 
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distributed with means ^ and y fi , variances a* and oj, and covariance o 

Note that this represents the simplest model for G(t) , which incorporates 
varying individual growth. While too simple to fully describe many growth 
processes, linear individual growth may be a reasonable analytic approximation 
over a short term even if long-term growth has a more complex forra^ " ^ 
As for the random component, the model assumes that 

E[R.(t)] = 0 
Var[R.(t)] = c 2 R (3) 
(i.e., fixed over subjects and time) and 

Cov[R.(t), ir] = Cov[R.(t), 6] = 0. 



'ihus we represent the observed pretest (t = t^ as 



\ 



\ 

\ 



W = Vi ( V + 6 i + (4) 



For convenience, let us define 

\ A. = a.(t 2 ) - a.Ctj), ( 5) 

where represents the time duration between pre- and posttest for subject 
i. Note that we are assuming that tj and t may differ across sutfjects, but 
are dropping the subscript i for notational convenience. 
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At the posttest (t = t^)$ in the absence of a treatment, we would have 

VV = Vi (t 2 } + 6 i + W 

= G i (t 1 ) ♦ ir.A. i R.(t 2 ), (6) 

where ir^A i represents the expected growth between pre- and postmeasure due 
solely to natural maturation. 

In the presence of a program, we assume that over the time interval 
tj to t 2 the treatment increases each subject's growth by an aroount v^, 
called the value added by the program. Thus we can represent the measured 
growth subject i achieves by time under an intervention as 

W = W + Vi + v i + R i (t 2 ) - C7) 

Under this model, the treatment effect is fully described by the dis- 
tribution of the v^. We assume that v is a random variable with mean y v , 
and variance a*. Normally, we Lre interested in a summary measure of the 
treatment effect. This suggests that we estimate y y , the average of the 

individual treatment effects. 

During the period between the pre- and post-measure, the observed 
change in the treatment group is Y'(t 2 ) - YCtj). The expected growth under 
the model is y A. If we knew the value of u , a natural estimator of u 

7T 7T V 

would be 

3o 
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Bryk et al. (1980) have shown, under the assumption that a and 6 are 
independent of aft^ and that v and A are independent, that the ordinary 
least squares regression of Y^) on a(tj) yields an unbiased estimate of 
u^, and as a result, 

V- Y (t 2 ) - Y (tp - (9) 
represents an unbiased estimate of u 

v 

While Bryk et al. (1980) do not derive an estimate of the standard 
error of V, they suggest the use of the jackknife technique (described i n 
Chapter 8 of Hosteller and Tukey, 1977). to provide both a test statistic 
and standard error of V. 

EXTENDING THE BASIC VALUE-ADDED MODEL: INCORPORATING BACKGROUND VARIABLES 

130 r he previous-^ecUon-r- -we-a^sumed that--the-gro\rth-ra1re -paramerer — r 

' i ' 

was a random variable that characterized each individual's determinants of 
growth on some outcome dimension of interest. Ke have implicitly assumed 
that tt. is unmeasurable. One obvious alternative is to consider models 
that incorporate additional background information besides age. We introduce 
in this section a model in which the individual growth rate t:. is represented 
as a function of measurable variables that stand in proxy for environmental 
and constitutional factors determining the individual's growth rate. 
Model Specification 

We -assume that each individual's growth rate can be represented as a 
linear function of measurable background variables: 
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J 




CIO) 



where 



is the value on the jth background variable for subject i, 
6 represents a vector of coefficients, 

represents unmeasured determinants of individual growth rate, and 
j = l ... J variables; i*l ... n subjects. 



We assume .further that 



Equations 10 and 11 constitute our model for the individual growth rate 
parameter. We are assuming here that the differences in tt. among individuals 
can, at least partially, be expressed as a function of measurable variables. 



In particular, since participants will have different values for the vector 
of background variables, X, different individual valites of it. are predicted. 



Substituting Equation 10 into our basic model, specified by Equation 4, 
we obtain 



Var(e. |X.) = o* 



and Cov(e. , X. ) = 0. 



(11) 



\ 



ERLC 



40 



-37- 



J 

Y.CtO = p 6 + eoa.Cti) + Z 8 X a.UJ + e* {12) 

j=l * 3 

where 

e* = R.(ti) + e.a.(ti) + (6. - p fi ) . (13) 

Since this is a linear function of the parameters 6, the simplest approach 
is to apply ordinary least squares (OLS) to the model in Equation 12 . How- 
ever, OLS will yield unbiased estimators only if 

E(e*|a(ti), X) = 0 " (14) 

Bryk et al . (1980) have shown that if we assume for any given value of X 
that a(ti) is uncorrelated with both tt and 6, then 

E(e*|a(ti) f X) - 0, (15) 

and OLS will therefore produce an unbiased estimate of 6. 

To understand this condition intuitively, let us consider a simpler 
(and less efficient) way that background information may be employed* If 
we simply stratified the data to create groups homogeneous in terns of 
their X values, we could then carry out separate value-added analyses using 
the basic procedure derived earlier for each group and combine the results 
by averaging across groups, {k simple version of this analysis was illustrated 
earlier in the main body of the resource book.) However, for the analysis 
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on any group to produce an unbiased estimate of the basic model 
assumption (i.e. that tt and 6 are independent of a(tj) and that tt and 6 
are- independent) must hold for that group. That is, it must hold con- 
ditionally on X. 

1 To apply the least squares estimation procedure (following Equation 
12) we regress the pretest, Y(t : ) on age and the first order interactions 
of age and the background variables. From the 6 coefficients we can now 
estimate an individual growth rate, that is, 



/s J /s 

it. = e 3 + z e.x. . (i6) 
1 j=i J l J 



and with these we can now calculate an average value-added, Vq, where 



V 5 ■ Y 2 " Y l " n^iV (17) 



Under cur assumptions we can easily demonstrate that 



E(VJ) = U v . (18) 

So, we obtain an unbiased estimator of y . 

v 

If the multiple correlation between tt and X is substantial * this 
estimator V~ should be more precise than t\e basic V. An exact expression 
for the variance, however, is rather complex, since it involves the co- 
variances among the estimated 6^'s, which depend on the values of both 

a(ti) and X. Similarly, an expression for Var(Vj) would depend on the unknown 
distributions for 6 and 6. 
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A Day Care Study Application 

As an illustration of how Vg can be applied, let us consider a small 
subset of the analyses performed as part of the National Day Care Study CNDCS) . 
The Preschool Inventory (PSI) was administered to a large sample of day care 
participants. A score for each child was recorded in the early fall (pretest) 
and again in mid-spring (posttest). In addition, a set of background informa- 
tion on the child and family was collected. 

One question of interest in the NDCS concerned the effects of different 
day care centers on the cognitive development (as reflected on the PSI) of 
children in these centers. The researchers wished to determine the average 
increment to a child's PSI score above. that resulting from natural maturation. 
For technical reasons, irrelevant to this example, a transformation of the 
PSI score was actually used as the outcome measure. 

Using the approach introduced above, we must fiirst estimate 0 coefficients 
for the individual growth rate model as specified by Equation 12. ,In the 
NDCS, four variables seemed most important on the. basis of theoretical knowl- 
edge and preliminary analyses. These were: 

X. = child's sex (1 = male; -1 = female), / 

1 i 

i 
I 

X = child's race (1 = black; -1 = white), / 

2 ( 

Xj = mother's education (1 = more than 12 years; -1 = 12 years or less), 

X^ = public assistance (1 = receives; -1 = does not receive). 

The regression of^ PSI pretest scores against age and the first-order 
interactions between age and these variables prcduced 
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Y(ti) = -.4086 + .2942a 1 (ti) - .0065a 1 (t)X 1 - .0345a 1 (t)X 2 
+ .oma^tJXj - .0167a 1 (t)X 4 . 

So the estimated value for any individual was given by 

J = .2942 - .0065X. - .0345X 2 + .0123X 3 - .01o7X 4 . 

To interpret this equation, consider two illustrative individuals, erne a 
"disadvantaged" child and the other a "middle-class" child. The individual 
participant data and resulting growth rate estimates are shown in Table A,l 

For Case I (disadvantaged child) : 

ij = .2942 .- .0065(1) - .0345(1) + ,0123(-1) - .0167(1) 

TTj = 0.2242 
For Case II (middle class child): 

* n = .2942-- .0065(1) - .0345(-l) + .0125(i: - .0167(-1) 

TTjj = 0.3642 

In general, these results are consistent with what we might expect. 

/*\ 

The estimated growth rate for the "disadvantaged child/ 1 tt t = .22, Is 
considerably lower than that for the more "advantaged child," r,^ = .36. 

Moreover, because the background variables reflec. group membership 
in this example, the 8 coefficients have a very simple interpretation. 
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Table A.l. Illustrative Data for Two Children 
in the National Day Care Study 



Variables 

PSI posttest 
(transformed metric) 

PSI pretest 
(transformed metric) 



Disadvantaged Child 



Sex, X 



1 



Race, X 2 

Mother f s Education, X. 
Public Assistance, X, 



.4000 



,2000 



1 (Male) 

C 

1 (Black) 
-1 ( <^ 12 years) 

1 (Yes) 
.36 (Years) 



Middle Class Child 

« 

.6000 

.4000 

-1 (Female) 
-1 (White 
1 ( > 12 years) 
-1 (No) 
.43 (Years) 



If we think of 6o as a general growth rate for the total sample, then each 

reflects an adjustment (either up or down) on the base rate associated 
with membership in come subgroup of the total sample. For example, for a 
male child is reduced by .0065, while for a female child it is raised by 
the same amount. Being female, then, corresponds to an increase of .0065 
over the average. We should note, however, that this straightforward inter- 
pretation represents only a description of the analytic model. No causal 
inferences are intended. 

We are now in a position to calculate Vg foa: an individual day care 
center. We compute values of tt. and for each participant. From Equation 
17 an overall estimate of \i is obtained. In this example V"2 = .172. 
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Note that since the background variables are discrete, we could have 
estimated a separate y^ for each subgroup of cases within the sample. In 
particular, we could have stratified the sample using the four independent 
variables, creating a 2 4 design, and then applied the basic value-added 
estimator, V, within each of the cells in the design. To be effective, this 
approach requires that the sample size within each cell be sufficiently 
large to generate a stable value for y^. The linear model approach illustrated, 
above exploits the data in a more efficient way, 
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APPENDIX B 



EFFECT OF SELECTION ON THE PRETEST ON ESTIMATING A 
NATURAL GROWTH RATE FROM THE 
SIMPLE REGRESSION OF PRETEST ON AGE 
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This appendix discusses the problems involved in estimating a regression 
coefficient based on a data set in which the observations form a non-random, 
truncated sample. This problen arises in applications of the value-added 
method to situations where the pretest score was used as a selection criteria. 
Thus children who did well on the pretest did not enter the program and are 
not part of the data set. Since children usually do better with age, this 
selection process will mean that fewer older children and more younger children 
will enter the program. This differential selection procedure with regard 
to age creates a situation in which the least-squares regression of pretest 
on age will result in a biased estimate of the natural growth rate occurring 
in the absence of intervention. 

Fortunately, the truncated dependent variable problem also occurs in 
econometrics. This appendix summarizes some of the findings of Hausman and 
Wise (1977), and applies them to the value-added situation, 

THE BASIC PROBLEM 

We begin by defining some variables and stating the value-added model: 

i=l,. . . ,n 
i=l,. . . ,n 



Y.(t^) = subject i ! s pretest score 
a^Ctj) = subject i ! s age at pretest 
8 = overall grcwth rate 



R.(t^) = subject i ! s random component 
1 of growth 
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Then the model is: 



where 



R^Ctj) - n(0, a*) and is independent over subjects i, 



and independent from a^Ctj) 



However, since any child scoring better than some cut-off, (or, more gene- 
rally, L i# where might be related to age), is not included, we can also 
say: 



if Y. < L. we include indivic.ual i, or 



if Y. > L. we exclude individual i 

l l 



For the most common case, where all are equal, Figure B,l illustrates 
the truncation problem. Here the solid line represents the underlying true 
relationship between age and test score in the population. If we screen 
children of say ages'aj, a 2 , and a^, we see that although the population, 
represented by dots, varies about this true line, we will accept into the 
program only those individuals with scorer below the cut-off, L, whose dots 
are circled. Since regressing test score on age for just the program sample 
involve^ only the circled dots, the broken line represents the resulting 
estimate of the growth trajectory. Thus, we underestimate the natural growth 
rate. If we underestimate the natural growth rate, however, we will under- 
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true growth 
trajectory, 3 




Age 



Figure B.l. Effects of Sample Truncation on Estimated Regression Line . 



estimate the expected gain over the program period, and thus overestimate 
the value-added by the program. 

From Figure B.l we can see that the least-squares slope, 3, will always 
be flatter than the true S (for positive 3, 6 fill be smaller than 6). For 
a given set of ages, the amount of the bias deper.ds on three things: the 
value of L, the true growth rate 3, and the variability of test scores around 
the growth curve, a 2 . As L increases, the bias decreases, as 3 decreases,^ 
the bias decreases; as c 2 decreases, the bias decreases. 
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Hausman and Wise derive an exact formula for the bias. This formula 
depends on the assumption of normality of errors, R. . First, they define 



d i = L i - fajcy 

o 

This is the standardized distance from E(Y. !a.) to the cut-off for age a.. 

i 1 1 & 1 

Now, for large samples, the bias of 6 is given in the following formula 

n ■ i Cti)«d i ) 
6 = true 6 • I 

3 = 1 1 

where $ is the standard normal probability density function, and $ is the 
standard normal distribution function. 

From Figure B.l we can also see that a lower cut-off would also bias 
the estimated slope downward, as would the operation of both upper and 
lower cut-offs together.' Hausman and Wise do not consider this problem, 
but the bias equation would presumably be as above, with an additional third 
term adding in the probability lost below the lower cut-off as well. 

As for the multivariate case, where more than one 6 are being estimated, 
the bias of individual parameter estimates cannot be determined a priori; but 
in general one should expect least squares estimates to be biased towards zero. 
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SOME ALTERNATIVE SOLUTIONS 

In response to this problem, Hausman and Wise developed a caximum likeli- 
hood solution for estimating B from truncated Y data, and. present an illus- 
tration of its application, ' The key assumption in the maximum likelihood 
estimation procedure is that the pretest score ^(tj) follow a conditional 
normal distribution with mean Ba^ (t ^ ) , Gl enday (1 978) documents a general 
computer program which can be used for this situation,* 

In terms of less technical (and also less efficient) ways to deal with 
the bias, there are several possibilities. If pretest and age data were 
collecced on a larger population from which the program sample was eventually 
drawn, the simple regression of pretest on age for the larger population should 
yield an unbiased estimate of the natural growth rate to be used as the basis 
for the value added estimate. Another possibility is to use least squares, 
but to limit consideration to observations that fall within some restricted 
range. For example, use only values of age for which the expected value of 
Y given age is well below the truncation point. Still another possiMlity is 
to consider only observations having values of the dependent variable well 
below the truncation level. Again, this may lead to observations that are 
restricted to lie within a limited range. 



A copy of this demonstration is available at your regional technical 
assistance center, as well as information on how to procure a copy of 
the computer program. 
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